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DoD  Weapon  Systems  Acquisition 
Continues  To  Be  High  Risk  Area 


While  DOD’s  acquisition  process  has  produced  the  best  weapons  in  the 
world,  it  also  yields  undesirable  consequences  in  weapon  system 
programs  -  cost  increases,  schedule  delays,  and  performance  shortfalls 

Problems  occur  because  weapon  programs  do  not  capture  early  on  the 
requisite  knowledge  that  is  needed  to  effectively  manage  risks 

Programs  move  forward  with  unrealistic  cost  and  schedules  estimates, 
lack  clearly  defined  and  stable  requirements,  use  immature  technologies, 
and  fail  to  solidify  design  and  manufacturing  processes  at  appropriate 
junctures  in  development 

As  a  result,  programs  require  more  resources  than  planned,  the  buying 
power  of  the  defense  dollar  is  reduced,  and  funds  are  not  available  for 
other  competing  needs 


Paul  Francis ,  Director 
Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
May  18,  2004 


GAO 

Accountability  *  Integrity  *  Reliability 
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Problem  Statement 


A  statement  of  organizational  process  maturity  or  capability  level 
does  not  guarantee  performance  to  that  same  level  of  proficiency 
on  an  individual  project 

Most  DoD  contractors  claim  high  maturity/capability  levels,  yet 
from  the  perspective  of  the  acquirer,  systems  engineering  and 
program  management  practices  are  severely  lacking 

Teaming  arrangements  further  cloud  the  issue  of  process 
execution  and  proficiency 

Associated  problems  may  not  be  evident  until  significant  cost, 
schedule,  or  performance  objectives  have  been  missed  at  a  late 
point  in  the  program  where  corrective  actions  are  very  costly 


How  can  the  acquirer  gain  the  necessary  insight  into  process 
execution  and  proficiency  as  well  as  reinforce  desired  behaviors? 
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Development  Challenges 


Huge  system/software  engineering  endeavors  in 
aircraft,  space  vehicles,  ground  systems,  C2,  C4ISR, 
battle  mgt,  etc 

Significant  risks  for  acquisition  programs 

•  Multi-million  SLOC  programs;  “hybrid”  systems 
combining  legacy  re-use,  COTS,  new  development 

•  Multi-contractor  teams  using  different  processes; 
Dispersed  engineering  &  development  locations 

•  New  technologies  create  opportunities/challenges; 
products  change/evolve 

•  Business/operational  needs  change  -  often  faster 
than  full  system  capability  can  be  implemented 

•  Contractor  and  Acquirer  skills  shortfalls;  Cost  and 
schedule  constraints 
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Acquirers’  Concerns 

Appraisals  are  conducted  just  for  the  “ratings” 

Appraisal  team/lead  objectivity  and  expertise  is 
questionable,  especially  with  high  maturity  appraisals 

Appraisals  are  conducted  on  dissimilar  projects 
and/or  in  unrelated  organizations/sites 

The  effect  of  teaming  arrangements  on  project 
process  performance  is  not  accounted  for 

There  is  a  lack  of  insight  into  contractor  process 
execution  on  specific  projects 

The  effect  of  process  immaturity  in  acquiring 
organizations 
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High  Maturity  ^ 
Organizations  VjF 


Maturity  Levels  are 
indicators  of  organizational 
potential  performance. 


They  describe  how  the  next 
project  may  perform  based 
on  a  sampling  of  existing 
projects. 


Maturity  Levels  reside  at  the 
organizational  level  and  are 
not  an  indication  of  how  an 
individual  project  is 
performing. 
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High  Maturity 
Projects 


appraisals  of 

THESE  ... 
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A  means  to  assess  developer  process  execution  on  their 
project 

•  Process  performance 

•  Process  capability 

•  Process  adherence 

A  means  to  assess  project  process  execution  across  large, 
distanced,  and  diverse  teams 

A  means  to  assess  all  technical  and  project  management 
process  areas 

•  Engineering 

•  Project  Management 

•  Financial  Management 

•  Support  Processes 

A  means  to  assess  project  risks  driven  by  developer 
process  execution 

A  means  to  assess  the  potential  process  execution  risks  in 
source  selection 
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Standard  CMMI  Appraisal  Method 
Process  Improvement  (SCAMPI) 


Benchmarking  and 
Baselining 


Tailorable  Assessment 
“Mini"  Appraisal 
as  executed 


Maturity  or  Capability 
ratings 


> 


Cost/Duration 
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Assessment  Alternatives  Mapped  to 
Acquirer  Needs 


SCAMPI  Method  A 

Institutionalization 
Organizational  focus 
Rigorous,  expensive 
Ratings 


SCAMPI  Method  B 

Deployment  and  execution 
Evidence  of  implementation 
What  they  are  doing 

SCAMPI  Method  C 

Approach 
Plan  for  execution 
What  they  will  do 


Resource  Intensive 
Limited  utility  for  source 
selection 


Contract  Monitoring 
Competitive  Downselect 


Contract  Monitoring 
Source  Selection 
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PIER  Ground  Rules 


Use  the  process  model  -  CM  Ml 

•  Interview  questions  based  on  model 
Appraisal  of  process  performance  and  adherence 

Focus  on  risk  assessment  -  risks  associated  with  process 
performance,  adherence,  and  capability 

Observe  strict  confidentiality 

•  Results  not  attributable  to  individuals  or  interview 
groups 

Approach  SCAMPI  collaboratively 

Results  in  actionable  findings  by  Program  Office  and/or 
Contractor 
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What  a  PIER  Is  and  Is  Not 


Is  an  appraisal  (SCAMPI-B)  of  the  execution  of  and  the 
risks  associated  with  process  implementation  of  the 
entire  project  team,  including  sub-contractors 

Is  a  mechanism  to  provide  insight  into  the  operations  of 
a  specific  program  with  respect  to  the  organization’s 
published  maturity  level 

Is  not  a  formal  appraisal  (SCAMPI-A) 

No  rating 

No  multiple  projects  within  the  organization;  focus  on 
specific  project 

No  coverage  of  entire  life  cycle;  focus  on  phase  or  near- 
term  milestones 
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Phase  1 :  Plan  and  Prepare  for  the  Apprais 


Appraisal  Team/Lead 

•  Lead  assists  Sponsor  in  matching 
appraisal  w/business  objectives 

•  Lead  and  sponsor  agree  on  details 
of  appraisal  plan 

•  Lead  prepares  team  members 

•  Prepares  in-briefs  for  contractor 

•  Lead  performs  readiness  review 


Contractor 

•  Coordinates  with  the  PMO 

•  Identify  site  coordinator,  process 
lead,  and  non-voting  team  member 
(as  appropriate) 

•  Schedule  facilities  &  resources  to 
support  the  plan 

•  Collect  relevant  objective  evidence 

•  Prepare  project  to  support  PIER 
(schedule  interviews,  briefings, 
demos) 


SEI/MITRE/ESC 

•  Templates  and  guidance  on  plan 
preparation  &  PIER  execution 

•  Assist  PMO 

•  Model  scope  guidance 

•  Role  definition  guidance  for 
customer  &  contractor 

•  Security  procedures 


PMO 

•  Establishes  contractual 
arrangement 

•  Sponsors  appraisal 

•  Establishes  business  goals 

•  Coordinates  with  Contractor 

•  Coordinates  &  approves 
appraisal  plan 


page  15 


Carnegie  Mellon 

Software  Engineering  Institute 

Phase  2:  Conduct 


MITRE 

Appraisal 


Appraisal  Team/Lead 

•  Examines  objective  evidence 

•  Verifies  the  implementation  of  the 
organization’s  practices 

•  Validates  findings 

•  Generate  appraisal  results 

•  Lead  collects  lessons  learned 

•  Data  &  artifacts  are  appropriately 
archived  or  destroyed 


Contractor 

•  Provide  Project  context  briefing 

•  Ensure  contractor  responsibilities 
are  met  to  support  PIER  (briefings, 
interviewees,  demos) 

•  Support  additional  evidence 
requests  in  a  timely  manner 

•  Provide  PIER  team  facilities  in 
accordance  with  plan  requirements 


SEI/MITRE/ESC 

•  Evidence  collection  tool 

•  Findings  briefing  template  and 
guidance 

•  On-site  scheduling  guidance 

•  Guidance  for  preparation  and 
delivery  of  preliminary  findings 

•  Further  definition  of  customer  & 
contractor  roles 


PMO 

•  Coordinate  as  necessary  with 
Team  Lead  &  Contractor 

•  Resolve  any  appraisal 
execution  issues 
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Phase  3:  Report  Results 


Appraisal  Team/Lead 

•  Delivers  final  findings  to  customer 
&  to  contractor 

•  Reports  results  to  PIER  steward  as 
authorized  by  PMO 


Contractor 

•  Attend  outbrief 

•  Analyze  results  &  incorporate  into 
next  cycle  of  process  improvements 


SEI/MITRE/ESC 

•  Findings  template 

•  Final  findings  delivery  guidance 

•  Disclosure  statement  guidance 

•  ESC  PIER  Custodian  who 
retains  information  as 
observations  &  trends  without 
project  or  Contractor  attribution 


PMO 

•  Attends  outbrief 

•  Owns  final  findings  &  authorizes 
release  of  information  to  ESC 
PIER  Custodian 

•  Analyzes  results  &  integrates  into 
project  data  for  future  use  in 
award  fee,  program  reviews, 
improvement  initiatives,  etc. 
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Example 

Integrated  Supplier  Management 

Description:  ^ 

The  purpose  of  Integrated  Supplier  Management A  k  roactively 
identify  sources  of  products  that  may  be  use  ^  isOisythe 
project’s  requirements  and  to  manage  fa.  jpti^e Suppliers  while 
maintaining  a  cooperative  proje^-suppwrrelationship. 

Strengths: 

•  Proactive  enc^  ym  -4/vith  suppliers  on  early  Time  and 
Materia'0  si  fc^rffacts  expedited  the  creation  of  supplier 
rec  /ire^^ats  documents  and  SOWs  for  the  program. 

•  The  lack  of  monitoring  of  supplier  processes  may  cause 
execution  of  supplier  agreements  to  be  compromised.  (SG  2) 

•  The  Subcontract  Management  Plan  is  still  in  draft  form.  (GG  2) 
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Example 

Integrated  Supplier  Management 

Notes: 

•  Issues  with  critical  subcontractors  exist  and  ->n^  \ub Jo 
worked  ~  ' 


u 
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Example 

Project  Management  Specific  Practices 
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Appraisal  Team/Lead 

•  Ensure  team  composition  is  high- 
caliber 

•  Everyone  shares  work  load  based 
on  team  expectations 

•  Collect  lessons  learned  to  improve 
the  PIER  (SCAMPI-B)  process 


Contractor 

•  Participation 

•  Evidence  prepared  with  adequate 
time  to  review  prior  to  on-site 

•  Resolution  of  access  to  information 
issues  in  advance  of  on-site 

•  Relevant  personnel  provided  for 
interviews 

•  Act  on  findings 


MITRE 


SEI/MITRE/ESC 

•  Support  and  refine  PIER 
(SCAMPI-B)  method 

•  ESC  PIER  Custodian 

•  Work  with  PMO  to  ensure 
appropriate  interpretation  &  use 
of  findings 

•  Ensure  confidentiality  of  results 

•  Provide  updated  templates  and 
materials 


PMO 

•  Contractual  obligation 

•  Resources 

•  Act  on  findings 
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PIER  Lessons  Learned 


Appraisal  Team/Lead 

•  Have  consistent  approach  among 
team  leads  across  program  PIERs 

•  Train  team  on  appraisal  tools  and 
templates  prior  to  the  appraisal 

•  Complete  document  review  prior  to 
start  of  the  on-site 

•  Site  coordinator  needs  to  know  the 
CMMI  model 

•  Team  building  activities  critical 


Contractor 

•  Provides  feedback  on  process 
implementation 

•  Non-trivial  amount  of  effort  to 
support  the  PIER  activity 

•  Demonstration  of  organization’s 
commitment  to  CMMI 


SEI/MITRE/ESC 

•  Need  guidance  and  templates  to 
ensure  consistency  of  PIERs 

•  Capture  observations  and 
trends  to  isolate  potential 
systemic  problems 

•  Plan  PIERs  to  avoid  impacting 
peak  or  critical  activities 

•  Contractor  non-voting  team 
member  proved  very  valuable 

•  Credibility  counts! _ 

PMO 

•  Provides  confidence  in 
contractor  through  insight 

•  Process  proficiency  and 
execution  varies  substantially 

•  Basis  for  assessing  contractor 
responsiveness 

•  PMOs  must  work  internal 
processes  to  complement 
Contractor 
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ESC  Case  Studies 
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Joint  Environmental  Tool  Kit  (JET) 

*  Contract  monitoring 

*  Risk  identification 

*  Competitive  downselect 

KG-3X  Crypto  Modernization  Program 

*  Contract  monitoring 

*  Risk  identification 

*  Competitive  downselect 

Joint  Mission  Planning  System  (JMPS) 

*  Contract  monitoring 

*  Risk  identification 
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Expand  the  scope  of  PIERs 
Financial  Management 

Assess  execution  of  Earned  Value  Management 
System  (EVMS)  practices  especially  in  correlating 
product  maturity  and  performance  to  earned  value  - 
Technical  Performance  Measures  (TPMs) 

Product  Quality 

Product  quality  in  terms  of  technical  maturity  and 
product  performance  using  models,  simulations, 
prototypes,  and  early  functional  assessments 

May  integrate  with  the  Integrated  Baseline 
Review  (IBR)  process 
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Summary 

SCAMPI-A  appraisals  and  advertised  CMMI  ratings  do  not 
guarantee  that  individual  projects  will  perform  to  those 
levels  of  proficiency 

PIERs  have  validated  the  SCAMPI-B  Method  in  a  contract 
monitoring  context 

PIERS  provide  insight  into  project  process  execution, 
both  for  Program  Office  and  for  Contractor 

PIERS  demonstrate  program  office  commitment  to 
process 

PIERs  can  be  complementary  components  to  other 
reviews  and  technical  interchanges 

PIERs  can  be  used  to  support  award  fee  evaluations 
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Contact  Information 


Lorraine  Adams 

Software  Engineering  Institute 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 
412-268-7777 
ladams@sei.cmu.edu 

Fred  Schenker 

Software  Engineering  Institute 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 
412-268-9145 
ars@sei.cmu.edu 


Lynda  Rosa 

ESC  ACE  Chief  Engineer 
9  Eglin  Street 

Hanscom  AFB,  MA  01731-2100 
781-377-5398 


Dale  Swanson 

The  MITRE  Corporation 
202  Burlington  Road 
Bedford,  MA  01730-1420 
781-266-9195 
swaneed@mitre.org 
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